Model-Based Offline Reinforcement Learning with Local Misspecification

We present a model-based offline reinforcement learning policy performance lower bound that explicitly captures dynamics model misspecification and distribution mismatch we propose an empirical algorithm for optimal selection. Theoretically, prove novel safe improvement theorem by establishing pessimism approximations to the value function. Our key insight is jointly consider selecting over models policies: as long can accurately represent of state-action pairs visited given policy, it possible approximate particular policy. analyze our in LQR setting also show competitive previous bounds on selection across set D4RL tasks.